Revealing the Blind Spot of Sentence Encoder Evaluation by HEROS

Existing sentence textual similarity benchmark datasets only use a single number to summarize how similar the sentence encoder's decision is to humans'. However, it is unclear what kind of sentence pairs a sentence encoder (SE) would consider similar. Moreover, existing SE benchmarks mainly consider sentence pairs with low lexical overlap, so it is unclear how the SEs behave when two sentences have high lexical overlap. We introduce a high-quality SE diagnostic dataset, HEROS. HEROS is constructed by transforming an original sentence into a new sentence based on certain rules to form a \textit{minimal pair}, and the minimal pair has high lexical overlaps. The rules include replacing a word with a synonym, an antonym, a typo, a random word, and converting the original sentence into its negation. Different rules yield different subsets of HEROS. By systematically comparing the performance of over 60 supervised and unsupervised SEs on HEROS, we reveal that most unsupervised sentence encoders are insensitive to negation. We find the datasets used to train the SE are the main determinants of what kind of sentence pairs an SE considers similar. We also show that even if two SEs have similar performance on STS benchmarks, they can have very different behavior on HEROS. Our result reveals the blind spot of traditional STS benchmarks when evaluating SEs.Comment: ACL 2023 repl4nlp (representation learning for NLP) workshop poster paper. Dataset at https://huggingface.co/datasets/dcml0714/Hero

C2KD: Cross-Lingual Cross-Modal Knowledge Distillation for Multilingual Text-Video Retrieval

Multilingual text-video retrieval methods have improved significantly in recent years, but the performance for other languages lags behind English. We propose a Cross-Lingual Cross-Modal Knowledge Distillation method to improve multilingual text-video retrieval. Inspired by the fact that English text-video retrieval outperforms other languages, we train a student model using input text in different languages to match the cross-modal predictions from teacher models using input text in English. We propose a cross entropy based objective which forces the distribution over the student's text-video similarity scores to be similar to those of the teacher models. We introduce a new multilingual video dataset, Multi-YouCook2, by translating the English captions in the YouCook2 video dataset to 8 other languages. Our method improves multilingual text-video retrieval performance on Multi-YouCook2 and several other datasets such as Multi-MSRVTT and VATEX. We also conducted an analysis on the effectiveness of different multilingual text models as teachers

Natural Language Embedded Programs for Hybrid Language Symbolic Reasoning

How can we perform computations over natural language representations to solve tasks that require symbolic and numeric reasoning? We propose natural language embedded programs (NLEP) as a unifying framework for addressing math/symbolic reasoning, natural language understanding, and instruction following tasks. Our approach prompts a language model to generate full Python programs that define functions over data structures which contain natural language representations of structured knowledge. A Python interpreter then executes the generated code and prints the output. Despite using a task-general prompt, we find that this approach can improve upon strong baselines across a range of different tasks including math and symbolic reasoning, text classification, question answering, and instruction following. We further find the generated programs are often interpretable and enable post-hoc verification of the intermediate reasoning steps

SemStamp: A Semantic Watermark with Paraphrastic Robustness for Text Generation

Existing watermarking algorithms are vulnerable to paraphrase attacks because of their token-level design. To address this issue, we propose SemStamp, a robust sentence-level semantic watermarking algorithm based on locality-sensitive hashing (LSH), which partitions the semantic space of sentences. The algorithm encodes and LSH-hashes a candidate sentence generated by an LLM, and conducts sentence-level rejection sampling until the sampled sentence falls in watermarked partitions in the semantic embedding space. A margin-based constraint is used to enhance its robustness. To show the advantages of our algorithm, we propose a "bigram" paraphrase attack using the paraphrase that has the fewest bigram overlaps with the original sentence. This attack is shown to be effective against the existing token-level watermarking method. Experimental results show that our novel semantic watermark algorithm is not only more robust than the previous state-of-the-art method on both common and bigram paraphrase attacks, but also is better at preserving the quality of generation

Ciprofloxacin-resistant Salmonella enterica Typhimurium and Choleraesuis from Pigs to Humans, Taiwan

We evaluated the disk susceptibility data of 671 nontyphoid Salmonella isolates collected from different parts of Taiwan from March 2001 to August 2001 and 1,261 nontyphoid Salmonella isolates from the National Taiwan University Hospital from 1996 to 2001. Overall, ciprofloxacn resistance was found in 2.7% (18/671) of all nontyphoid Salmonella isolates, in 1.4% (5/347) of Salmonella enterica serotype Typhimurium and in 7.5% (8/107) in S. enterica serotype Choleraesuis nationwide. MICs of six newer fluoroquinolones were determined for the following isolates: 37 isolates of ciprofloxacin-resistant (human) S. enterica Typhimurium (N = 26) and Choleraesuis (N = 11), 10 isolates of ciprofloxacin-susceptible (MIC <1 μg/mL) (human) isolates of these two serotypes, and 15 swine isolates from S. enterica Choleraesuis (N = 13) and Typhmurium (N = 2) with reduced susceptibility to ciprofloxacin (MIC >0.12 μg/mL). Sequence analysis of the gryA, gyrB, parC, parE, and acrR genes, ciprofloxacin accumulation; and genotypes generated by pulsed-field gel electrophoresis with three restriction enzymes (SpeI, XbaI, and BlnI) were performed. All 26 S. enterica Typhimurium isolates from humans and pigs belonged to genotype I. For S. enterica Choleraesuis isolates, 91% (10/11) of human isolates and 54% (7/13) of swine isolates belonged to genotype B. These two genotypes isolates from humans all exhibited a high-level of resistance to ciprofloxacin (MIC 16–64 μg/mL). They had two-base substitutions in the gyrA gene at codons 83 (Ser83Phe) and 87 (Asp87Gly or Asp87Asn) and in the parC gene at codon 80 (Ser80Arg, Ser80Ile, or Ser84Lys). Our investigation documented that not only did these two S. enterica isolates have a high prevalence of ciprofloxacin resistance nationwide but also that some closely related ciprofloxacin-resistant strains are disseminated from pigs to humans